Software testing provides software developers with information about the quality of a software application. Typical testing methods involve investigation of different aspects of the software, such as tracing the execution path of software or finding software bugs (e.g., errors and failures). Software testing validates and verifies a software application by checking whether the software executes as expected and produces desired outputs. One form of software testing is referred to as “unit testing.” In unit testing, rather than testing the entire behavior of a whole application, a software developer tests individual components or “units” of the application to confirm that the specific behaviors specified for or required by such particular “units” have been implemented correctly. For example, in object-oriented programming, such as JAVA brand computer programs, a unit might be an individual method or a class.
In JAVA brand computer programs, JUnit, widely used by developers, is a common unit test framework for testing JAVA brand computer program applications. JUnit provides a developer a set of JAVA brand computer program library classes that enables the developer to easily write pieces of testing code that validate the behavior of particular JAVA brand computer program classes or features (e.g., methods, etc.) that the developer has separately written as part of his actual JAVA brand computer program application. For example, in his testing code, the developer might instantiate an instance or object for a JAVA brand computer program class (e.g., myCalculator=new Calculator( )) that he has recently written and then invoke a JUnit library command that enables the developer to confirm that a particular input to a method of the JAVA brand computer program object actually produces an expected output (e.g., assertEquals(50, myCalculator.add(45, 4))). NUnit is a similar unit-testing framework for another popular object-oriented programming language, C# (C-sharp).
Unit tests that are more complicated than the above can further utilize an additional “mock” framework, such as EasyMock. In general, a “mock” that is created by such a mock framework simulates the behavior of objects or methods that may interact with other objects whose behavior is the subject of testing. For example, a mock framework may enable a developer who has not yet written an Adder class that is used in a multiply method of his Calculator class to create a “mock” of the Adder object so that he can test his recently written multiply method, which actually uses an Adder object:
public int multiply(int a, int b) {int x;int output = b;Adder adder = new Adder( );for x = 1 to a−1 {output = adder.add(output, b);}return output;}In his testing code, since the developer has not yet written the Adder class, he could instantiate a “mock” object of the Adder class and then specify, for example, that the add( ) method of the Adder class (i) should output 4 if called as Adder.add(2, 2), and (ii) should output 6 if called as Adder.add(4, 2). As such, the developer, using both the mock and unit test frameworks, could then test whether the code implementing the multiply method (see above) produces an expected output for multiply(3, 2), namely 6 (e.g., assertEquals(6, myCalculator.multiply(3,2)).
As can be seen from the foregoing, the combination of a unit-test framework and a mock framework facilitates testing of object-oriented code. However, common mock frameworks have a significant restriction when used with multi-threaded applications. Such mock frameworks can execute tests on a single thread and do not provide any efficient way to test classes or features that involve multiple threads.
Testing frameworks for multi-threaded applications need to deal with at least two significant issues: (i) how to report failed tests that may occur in a variety of spawned threads, and (ii) how to define thread-specific tests. For example, if a main thread that runs the testing code spawns another thread (e.g., due to the nature of the objects being run in the testing code), upon a failure of a particular test of the testing code during execution of the spawned thread, the spawned thread may throws an exception as a result of the failed test. Under current mock frameworks, however, the main thread that is running the testing code cannot catch the exception and therefore the failed test is not properly reported by the main thread. Additionally, although current mocking frameworks provide the capability to specify “stubs” (e.g., specifying, for example, the behavior of a mock object to simulate real objects, such as specifying that a mock object for Adder that receives a call of Adder.add(2, 2) should return the value 4) and “expectations” (e.g., specifying, for example, that a sequence of invocations of mock objects should occur during execution of the testing code such as specifying that the Adder.add( ) method should be invoked twice if the testing code is myCalculator.multiply(3, 2)) in the testing code, such current mocking frameworks do not provide support to define such stubs or expectations for a specific thread or set of threads.
While the existing mocking frameworks bring many desirable features to unit testing, some issues remain unsolved for multi-threaded applications.